Shopify For Dummies by Paul Waddy

Author:Paul Waddy [Waddy, Paul]
Language: eng
Format: epub
ISBN: 9780730394501
Publisher: Wiley
Published: 2022-07-07T00:00:00+00:00

When can I start running AB tests?

You may feel frustrated if you don’t have enough traffic to AB test yet, because it sounds fun — and it is! But you need to wait until you’ve got enough traffic on your site, so if you’re just getting started then you may have a little longer to wait. How much is enough? The answer is, enough to get a statistically significant outcome from the test, and generally, the more the merrier.

In the Shopify blog article ‘The complete guide to AB testing’, Shanelle Mullin references an AB test calculator from Evan Miller that estimates the sample size needed to run a successful AB test. Miller’s calculator asks for your regular, or baseline, conversion rate, and the minimum detectable effect, which is the smallest acceptable improvement for the test to be a clear winner. A 15 per cent relative improvement in conversion rate is what is suggested, meaning you want your conversion rate to be lifted by greater than 15 per cent for the variant to be considered statistically significant — therefore, a win. Any improvement on conversion rate that has a relevant improvement of less than 15 per cent isn’t worth it, or at least you can’t be sure it’s a clear winning test. To be clear, a relative improvement of 15 per cent on a 5 per cent conversion rate is not 20 per cent, it is 5.75 per cent. You’re not talking about changing the conversion rate from 5 to 15; instead, you need to improve the conversion rate by 15 per cent.

So, if I’m running an online store with a conversion rate of 5 per cent and I want to detect if an AB test can deliver greater than a 15 per cent improvement on conversion rate, I need traffic of 13,533 during the AB test.

The length of the AB test is ideally a full business cycle — let’s call it 30 days. This allows for various activities to be normalized — such as the general time it takes for a customer to browse, browse again and then purchase; or external factors, such as positive or negative press, and sale periods.

Miller’s AB test sample size calculator can be found here: evanmiller.org/ab-testing/sample-size.html.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.